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We present a Monte Carlo analysis in terms of neutrino oscillations of the total rates measured in solar neutrino 
experiments in the framework of frequentist statistics. We show that the goodness of fit and the confidence level 
of the allowed regions in the space of the neutrino oscillation parameters are significantly overestimated in the 
standard method. We also present a calculation of exact allowed regions with correct frequentist coverage. We 
show that the exact VO, LMA and LOW regions are much larger than the standard ones and merge together 
giving an allowed band at large mixing angles for all Am^ > 10"^'' eV^. 



1. INTRODUCTION 

Solar neutrino experiments ||l| have observed a 
flux of neutrinos smaller than the one predicted 
by the Standard Solar Model (see, for example, 
Ref. |2|). Neutrino oscillations (see, for example, 
Ref. |3|) is widely considered to be the simplest 
and most attractive explanation of this anomaly. 
Assuming the simplest case of two-neutrino oscil- 
lations, the statistical analysis of solar neutrino 
data yields allowed regions for the oscillation pa- 
rameters t\rr? = m\ — ml, where toi and m2 arc 
the two neutrino masses, and tan^ 6, where 6 is 
the neutrino mixing angle. 

The allowed regions in the tan^ plane 
are usually determined through a least-square fit 
(see, for example, Refs.[Hj|]), in which the param- 
eters are estimated through the minimum of the 
function 



(1) 



where Nr- 



3 is the number of experimental 



data points, and r'^'^^^^ are the theoretical 

and experimental rates, respectively, and V is the 
covariance matrix that accounts for experimental 
and theoretical uncertainties The theo- 
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retical rates R 



(thr) 



and the covariance matrix V 



depend on the parameters Am^, tan^ 0. 

Usually, in the application of the least-squares 
method it is assumed that is distributed as 
a with iVcxp = 3 degrees of freedom, X^^^^^ is 



distributed as a x with A'e; 



1 degrees 



of freedom, and — X^-^^ has a distribution 
with A^par = 2 degrees of freedom (see, for exam- 
ple, Ref. ||9|,[l0|| ) . This would be correct if: 1) the 
theoretical rates depended linearly on the param- 
eters; 2) the errors of the differences between the 
theoretical and experimental rates were multinor- 
mally distributed with a constant covariance ma- 
trix. Actually, these requirements are not satis- 
fied. In particular, it is well-known that the theo- 
retical rates have a complicate dependence on the 
parameters. For example, in the simplest case of 
oscillations in vacuum the electron neutrino sur- 
vival probability depends on Am^ through a si- 
nusoidal function: 



1 - sin^ 20sin2 ( Am'^L/AE) 



(2) 



Moreover, the covariance matrix V is not con- 
stant, but depends on Am^ and tan^ 9 (see 
Refs.[H,|[0]), and the errors of the differences be- 
tween the theoretical and experimental rates are 
not multinormally distributed (this is due to the 
fact that each theoretical rate is given by the 
product of the neutrino flux times the experimen- 
tal cross section; even if the errors of the neu- 
trino flux and the errors of the experimental cross 
section are normally distributed, their product is 
not). 
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Since is not a x^, in order to perform a reli- 
able statistical analysis of the data it is necessary 
to calculate with a Monte Carlo the distribution 
of its minimum, X^-^^, which is the estimator of 
the parameters Am^, tan^ 6. 

In Section || we present the result of our Monte 
Carlo estimation of the goodness of fit and the 
confidence level (CL) of the standard allowed re- 
gions. In Section ^ we present a calculation of 
exact allowed region with correct frequentist cov- 
erage. A detailed explanation of our procedure 
and results has been presented in Ref. [||. 

2. MONTE CARLO GOODNESS OF FIT 
AND CONFIDENCE LEVELS 

The exact distribution of X^^^^ is determined 
by the true value of the parameters Am^, tan'^ 6, 
which are unknown. Nevertheless, it is possible 
to estimate the distribution assuming surrogates 
for the true value of the parameters. The most 
reasonable surrogates are the best-fit values Am^ 
and tan^ 9 (see, for example, Ref. [|lO| ) . Assuming 

Am^ , tan^ 9 as surrogates of the true values of the 
parameters, we generate Ns synthetic data sets 
which simulate Ng different independent sets of 
experiments. Using these sets we can estimate 
the goodness of fit and confidence levels of the 
standard allowed regions. 

Goodness of fit is the probability to find in a 
set of hypothetical repeated experiments a X^-^ 
larger than the one actually observed. 

From the rates analysis (with the 1999 data 
summarized in Ref.||^), we find that, constrain- 
ing the parameter in an area around the global 
minimum (which is the SMA region), the stan- 
dard goodness of fit is reliable. This is due to 
the fact that in the neighborhood of the global 
minimum the dependence of the theoretical rates 
from the parameters is approximately linear and 
the covariance matrix is almost constant. 

On the other hand, if the parameters are un- 
constrained (allowing all the MSW region with 
10-8 eV^ < Am^ < 10"'' eV^ and the VO region 
with 10"" eV^ < Am2 < 10"* eV^), the good- 
ness of fit is 40%, poorer than the 52% obtained 
with the standard method. The fit is worse! We 



conclude that the goodness of fit is overestimated 
by the standard procedure. 

From the synthetic data sets we can also esti- 
mate the confidence level of the standard allowed 
regions. We start from the definition of confidence 
level of interval: it is the fractional number of in- 
tervals obtained in repeated experiments which 
cover the true values of parameters. 

In the standard procedure the confidence inter- 
vals are determined by the condition 

X'<Xl,^ + Ax'{(3), (3) 

where Ax^(/3) is the value of such that the cu- 
mulative x^ distribution for a number of degrees 
of freedom equal to the number of parameters is 
equal to the confidence level /3. But this proce- 
dure would be correct only if X'^ were a x^- 

In our simulation, for a fixed /3, we calculate 
the standard allowed regions for each synthetic 
data set. Then we count how many of them cover 
the surrogates of true values of parameters. This 
fraction is our estimate Pmc of the true confidence 
level of the standard allowed regions at (3 CL. 

We found that, if the parameters are con- 
strained around the global minimum and there 
is only one standard P CL allowed region, its 
confidence level /3mc is approximately equal to 
/3. Instead, if the parameters are not constrained 
around the global minimum and there are several 
standard f3 CL allowed regions, their confidence 
level /3mc is significantly smaller than /3. For ex- 
ample, the Monte Carlo confidence level of the 
standard 90% CL allowed regions is only 86%. 

This result was expected, because in the case 
of a real x^ there is only one minimum that de- 
termines an elliptic allowed region, whereas 
has several local minima (especially in the vac- 
uum oscillation region). Therefore, in repeated 
experiments there is a higher probability that the 
global minimum falls far from the assumed sur- 
rogates Am^ , tan^ 9 of the true values of param- 
eters, leading to a lower probability that the al- 
lowed regions cover Am^, tan^ 9. 

The approach presented in this section allows 
to calculate only estimations of the goodness of 
fit and the confidence level of the standard al- 
lowed regions, because the calculation depends on 
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Figure 1. Allowed 90%, 95%, 99%, 99.73% CL regions in the MSW part of the tan^ 6'-Am2 plane. The 
gray areas are the allowed regions with exact frequentist coverage in the MSW region; the areas enclosed 
by the solid lines are the standard SMA, LMA and LOW allowed regions. 



the assumed surrogates of the true values of the 
parameters. In the next section we present the 
results of a calculation of allowed regions with 
exact confidence level, which is independent from 
the unknown true value of the parameters. 

3. EXACT ALLOWED REGIONS 

The construction of exact confidence intervals 
has been introduced by Neyman in 1937 (see, for 
example, Rcf . P,pl| ) . It guarantees that the re- 
sulting confidence intervals have correct frequen- 
tist coverage, i.e. they belong to a set of con- 
fidence intervals obtained with different or sim- 
ilar, real or hypothetical experiments that cover 
the true values of the parameters with the de- 
sired probability given by the chosen confidence 
level (see, for example, Ref. [|l2|). We apply this 
method in order to find confidence intervals with 
proper coverage for the neutrino oscillation pa- 
rameters. 

Starting with the choice of an appropriate es- 
timator of the parameter under investigation, for 



any possible value of the parameter one calculates 
an acceptance interval with probability /3, i.e. an 
interval of the estimator that contains 100/3% of 
the values of the estimator obtained in a large 
series of trials. 

Once the 100/3% acceptance interval for each 
possible value of the parameter is calculated, the 
100/3% confidence interval is simply composed by 
all the parameter values whose acceptance inter- 
val covers the measured value of the estimator. 

This procedure can be generahzed to the case 
of more parameters: the acceptance intervals 
(and also the confidence intervals) are multidi- 
mensional regions and could be composed by dis- 
joint subintervals. 

In the case of solar neutrino oscillations, we 
have two parameters, Ara^ and tan^ 6, estimated 
through the minimum of X^. We consider a grid 
with 5000 points in the MSW region and 6000 
points in the VO region. For each point of the 
grid we generate about 6.5 x 10^ synthetic data 
sets that allow to calculate the distribution of 
the estimator X^^^^ and the consequent accep- 
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Figure 2. Allowed 90%, 95%, 99%, 99.73% CL 
regions in the VO part of the tan^ plane. 
The gray areas are the allowed regions with exact 
frequentist coverage in the VO region; the areas 
enclosed by the solid lines are the standard VO 
allowed regions. 



tance regions. More details have been presented 
in Ref.||. 

Our results are shown in Figures ^ and |^, where 
we have plotted the 90%, 95%, 99%, 99.73% CL 
regions in the MSW and VO parts of the tan^ 9- 
Am^ plane. One can see that the exact allowed 
regions are much larger than the standard ones. 
In particular, the LMA, LOW and VO regions 
are connected, giving an allowed band at large 
mixing angles for all Am^ > 10"^*^ eV^. Only the 
standard SMA region is a good approximation of 
the exact one. 

4. CONCLUSIONS 

We have calculated with Monte Carlo the good- 
ness of fit and the confidence level of the standard 
allowed regions of the two-neutrino oscillation pa- 



rameters Am^ and tan^ 9 obtained from the fit of 
the total rates measured by solar neutrino exper- 
iments. 

As expected, we found that the standard 
method overestimates the goodness of fit and the 
confidence level of the standard allowed regions. 

Using Neyman's construction, we have calcu- 
lated exact allowed regions with correct frequen- 
tist coverage. We have shown that the exact 
VO, LMA and LOW regions are much larger 
than the standard ones and merge together giv- 
ing an allowed band at large mixing angles for all 
Am2 > 10-1" eV^ 
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